Semi-automated curation of protein subcellular localization: a text mining-based approach to Gene Ontology (GO) Cellular Component curation

Background: Manual curation of experimental data from the biomedical literature is an expensive and time-consuming endeavor. Nevertheless, most biological knowledge bases still rely heavily on manual curation for data extraction and entry. Text mining software that can semi- or fully automate information retrieval from the literature would thus provide a significant boost to manual curation efforts. Results: We employ the Textpresso category-based information retrieval and extraction system http://www.textpresso.org webcite, developed by WormBase to explore how Textpresso might improve the efficiency with which we manually curate C. elegans proteins to the Gene Ontology's Cellular Component Ontology. Using a training set of sentences that describe results of localization experiments in the published literature, we generated three new curation task-specific categories (Cellular Components, Assay Terms, and Verbs) containing words and phrases associated with reports of experimentally determined subcellular localization. We compared the results of manual curation to that of Textpresso queries that searched the full text of articles for sentences containing terms from each of the three new categories plus the name of a previously uncurated C. elegans protein, and found that Textpresso searches identified curatable papers with recall and precision rates of 79.1% and 61.8%, respectively (F-score of 69.5%), when compared to manual curation. Within those documents, Textpresso identified relevant sentences with recall and precision rates of 30.3% and 80.1% (F-score of 44.0%). From returned sentences, curators were able to make 66.2% of all possible experimentally supported GO Cellular Component annotations with 97.3% precision (F-score of 78.8%). Measuring the relative efficiencies of Textpresso-based versus manual curation we find that Textpresso has the potential to increase curation efficiency by at least 8-fold, and perhaps as much as 15-fold, given differences in individual curatorial speed. Conclusion: Textpresso is an effective tool for improving the efficiency of manual, experimentally based curation. Incorporating a Textpresso-based Cellular Component curation pipeline at WormBase has allowed us to transition from strictly manual curation of this data type to a more efficient pipeline of computer-assisted validation. Continued development of curation task-specific Textpresso categories will provide an invaluable resource for genomics databases that rely heavily on manual curation

Text mining meets community curation: a newly designed curation platform to improve author experience and participation at WormBase

Biological knowledgebases rely on expert biocuration of the research literature to maintain up-to-date collections of data organized in machine-readable form. To enter information into knowledgebases, curators need to follow three steps: (i) identify papers containing relevant data, a process called triaging; (ii) recognize named entities; and (iii) extract and curate data in accordance with the underlying data models. WormBase (WB), the authoritative repository for research data on Caenorhabditis elegans and other nematodes, uses text mining (TM) to semi-automate its curation pipeline. In addition, WB engages its community, via an Author First Pass (AFP) system, to help recognize entities and classify data types in their recently published papers. In this paper, we present a new WB AFP system that combines TM and AFP into a single application to enhance community curation. The system employs string-searching algorithms and statistical methods (e.g. support vector machines (SVMs)) to extract biological entities and classify data types, and it presents the results to authors in a web form where they validate the extracted information, rather than enter it de novo as the previous form required. With this new system, we lessen the burden for authors, while at the same time receive valuable feedback on the performance of our TM tools. The new user interface also links out to specific structured data submission forms, e.g. for phenotype or expression pattern data, giving the authors the opportunity to contribute a more detailed curation that can be incorporated into WB with minimal curator review. Our approach is generalizable and could be applied to additional knowledgebases that would like to engage their user community in assisting with the curation. In the five months succeeding the launch of the new system, the response rate has been comparable with that of the previous AFP version, but the quality and quantity of the data received has greatly improved

Tissue enrichment analysis for C. elegans genomics

Background: Over the last ten years, there has been explosive development in methods for measuring gene expression. These methods can identify thousands of genes altered between conditions, but understanding these datasets and forming hypotheses based on them remains challenging. One way to analyze these datasets is to associate ontologies (hierarchical, descriptive vocabularies with controlled relations between terms) with genes and to look for enrichment of specific terms. Although Gene Ontology (GO) is available for Caenorhabditis elegans, it does not include anatomical information. Results: We have developed a tool for identifying enrichment of C. elegans tissues among gene sets and generated a website GUI where users can access this tool. Since a common drawback to ontology enrichment analyses is its verbosity, we developed a very simple filtering algorithm to reduce the ontology size by an order of magnitude. We adjusted these filters and validated our tool using a set of 30 gold standards from Expression Cluster data in WormBase. We show our tool can even discriminate between embryonic and larval tissues and can even identify tissues down to the single-cell level. We used our tool to identify multiple neuronal tissues that are down-regulated due to pathogen infection in C. elegans. Conclusions: Our Tissue Enrichment Analysis (TEA) can be found within WormBase, and can be downloaded using Python’s standard pip installer. It tests a slimmed-down C. elegans tissue ontology for enrichment of specific terms and provides users with a text and graphic representation of the results

Two new functions in the WormBase Enrichment Suite

Genome-wide experiments routinely generate large amounts of data that can be hard to interpret biologically. A common approach to interpreting these results is to employ enrichment analyses of controlled languages, known as ontologies, that describe various biological parameters such as gene molecular or biological function. In C. elegans, three distinct ontologies, the Gene Ontology (GO), Anatomy Ontology (AO), and the Worm Phenotype Ontology (WPO) are used to annotate gene function, expression and phenotype, respectively (Ashburner et al. 2000; Lee and Sternberg, 2003; Schindelman et al. 2011). Previously, we developed software to test datasets for enrichment of anatomical terms, called the Tissue Enrichment Analysis (TEA) tool (Angeles-Albores and Sternberg, 2016). Using the same hypergeometric statistical method, we extend enrichment testing to include WPO and GO, offering a unified approach to enrichment testing in C. elegans. The WormBase Enrichment Suite can be accessed via a user-friendly interface at http://www.wormbase.org/tools/enrichment/tea/tea.cgi. To validate the tools, we analyzed a previously published extracellular vesicle (EV)-releasing neuron (EVN) signature gene set derived from dissociated ciliated EV neurons (Wang et al. 2015) using WormBase Enrichment Suite based on the WS262 WormBase release. TEA correctly identified the CEM, hook sensillum and IL2 neuron as enriched tissues. The top phenotype associated with the EVN signature was chemosensory behavior. Gene Ontology enrichment analysis showed that cell projection and cell body were the most enriched cellular components in this gene set, followed by the biological processes neuropeptide signaling pathway and vesicle localization further down. The tutorial script used to generate the figure above can be viewed at: https://github.com/dangeles/TissueEnrichmentAnalysis/blob/master/tutorial/Tutorial.ipynb The addition of Gene Enrichment Analysis (GEA) and Phenotype Enrichment Analysis (PEA) to WormBase marks an important step towards a unified set of analyses that can help researchers to understand genomic datasets. These enrichment analyses will allow the community to fully benefit from the data curation ongoing at WormBase

An overview of the BioCreative 2012 Workshop Track III: interactive text mining task

In many databases, biocuration primarily involves literature curation, which usually involves retrieving relevant articles, extracting information that will translate into annotations and identifying new incoming literature. As the volume of biological literature increases, the use of text mining to assist in biocuration becomes increasingly relevant. A number of groups have developed tools for text mining from a computer science/linguistics perspective, and there are many initiatives to curate some aspect of biology from the literature. Some biocuration efforts already make use of a text mining tool, but there have not been many broad-based systematic efforts to study which aspects of a text mining tool contribute to its usefulness for a curation task. Here, we report on an effort to bring together text mining tool developers and database biocurators to test the utility and usability of tools. Six text mining systems presenting diverse biocuration tasks participated in a formal evaluation, and appropriate biocurators were recruited for testing. The performance results from this evaluation indicate that some of the systems were able to improve efficiency of curation by speeding up the curation task significantly (∼1.7- to 2.5-fold) over manual curation. In addition, some of the systems were able to improve annotation accuracy when compared with the performance on the manually curated set. In terms of inter-annotator agreement, the factors that contributed to significant differences for some of the systems included the expertise of the biocurator on the given curation task, the inherent difficulty of the curation and attention to annotation guidelines. After the task, annotators were asked to complete a survey to help identify strengths and weaknesses of the various systems. The analysis of this survey highlights how important task completion is to the biocurators’ overall experience of a system, regardless of the system’s high score on design, learnability and usability. In addition, strategies to refine the annotation guidelines and systems documentation, to adapt the tools to the needs and query types the end user might have and to evaluate performance in terms of efficiency, user interface, result export and traditional evaluation metrics have been analyzed during this task. This analysis will help to plan for a more intense study in BioCreative IV

2018 Update on Protein-Protein Interaction Data in WormBase

Protein interaction is an important data type to understand the biological function of proteins involved in the interaction, and helps researchers to deduce the biological nature of unknown proteins from the well-characterized functions of their interaction partners. High-throughput studies, coupled with the aggregation of individual experiments, provides a global 'snapshot' of the protein interactions occurring at all levels of biological processes or circumstances. This snapshot of the interaction network, the interactome, is important to understand the overall events up to the level of comparison between species or pathway simulation, or to find new factors yet undefined in the processes, or to add details to the biological processes and pathways. As of September 2018, WormBase (www.wormbase.org) (Lee et al. 2018) contains 28,279 physical protein-protein interactions for the roundworm Caenorhabditis elegans. Among these, 1500 protein-protein interactions have been curated by BioGRID as a collaboration with WormBase. Within the data set, 17,990 protein-protein interactions are unique, and 6,079 unique genes are involved in these interactions. In order to visualize the overall interaction map, a network diagram for all the unique interactions was generated by using the ‘Cytoscape’ program, version 3.6.1 (Shannon et al. 2003) (Figure 1A). These numbers represent a 108% increase in the number of interaction annotations since last year, 2017. These interaction data were curated from 1,251 peer-reviewed papers, which were selected from the literature by ‘Textpresso Central’ using automatic SVM (Support Vector Machine)-based text mining approaches (Fang et al. 2012; Müller et al. 2018) and manual verification. Compared to other databases providing C. elegans protein-protein interaction, WormBase now presents the largest data set, which has 1.72-fold more interaction annotations than IMEx (Orchard et al. 2012) and 4.51-fold more than BioGRID (Chatr-Aryamontri et al. 2017) (Figure 1B). Most significantly, WormBase now houses the complete protein interaction data from almost all of the C. elegans literature published from 1993 to 2018. The data sets presented at IMEx and BioGRID are annotated from 253 and 174 papers, respectively. All the physical interaction data in WormBase are supported by experimental evidence from original research papers. The statistics of the detection methods used as experimental evidence are shown in Figure 1C. The majority of the interaction data came from high throughput analysis such as large-scale yeast two-hybrid assays or mass-spectrometry, however, a significant portion of the data (13.1%) are supported by more direct detection methods using small-scale, low throughput methods such as co-immunoprecipitation or co-crystallography (Figure 1C). In WormBase, protein-protein interaction data can be found as a subclass of physical interaction data in the ‘Interactions widget’ on the gene report page. The Interactions widget provides all types of interaction data related to the gene of interest, such as physical, genetic, regulatory, and predicted interactions. All the interaction data are represented together in a graph created with ‘Cytoscape.js’ and a table. In the table, the gene names of interaction partners (bait-target) in the interaction are displayed along with the publication. The interaction details including the detection method are also captured in the summary and the remark field in the Interactions page. Users can query the data by using the search bar on the WormBase front page or download all the available data files from the WormBase FTP site (ftp://ftp.wormbase.org/pub/wormbase/releases/current-production-release/species/c_elegans/PRJNA13758 /annotation/c_elegans.PRJNA13758.WSXXX.interactions.txt.gz, where WSXXX is the database version release, like “WS267”). All the interaction data in WormBase will be available soon at the new information resource for multiple model organisms, the Alliance of Genome Resources (https://www.alliancegenome.org/). This site will integrate all the interaction data from human and from model organisms C. elegans, budding yeast (Saccharomyces cerevisiae), fruit fly (Drosophila melanogaster), zebrafish (Danio rerio), mouse (Mus musculus) and rat (Rattus norvegicus). Integrated views of interaction data from diverse model organisms will be extremely helpful to build interaction databases for species-to-species comparison, and to establish a disease model quickly based on the database. For the most efficient analysis of the interaction data in WormBase, we are now working on developing a new ‘Venn diagram tool’ and integrating the ‘Gene Set Enrichment Analysis tool’ (https://wormbase.org/tools/enrichment/tea/tea.cgi) into the Interactions widget. We will continue to curate other types of macro-molecular interactions including protein-DNA, protein-RNA and RNA-RNA interactions, as well as newly reported protein-protein interaction data to serve our research community

Textpresso - an Information Retrieval and Extraction System for Biological Literature

We developed an information retrieval and extraction system that processes the full text of biological papers. The system, called Textpresso, separates text into sentences, labels words and phrases according to an ontology (an organized lexicon), and allows queries to be performed on a database of labeled sentences. The current ontology comprises approximately one hundred categories of terms, such as "gene", "regulation", "human disease", "brain area" etc., and also contains main Gene Ontology (GO) categories. Extraction of particular biological facts, such as gene-­gene interactions, or the curation of GO cellular components, can be accelerated significantly by ontologies, with Textpresso automatically performing nearly as well as expert curators to identify sentences. Search engine for four literatures, C. elegans, Drosophila, Arabidopsis and Neuroscience have been established by us, and thirteen systems for other literatures have been developed by other groups around the world. Currently, our four systems contain 112,000 papers with 40 million sentences, all systems worldwide contain 190,000 papers with approximately 65 million sentences

2018 Update on Protein-Protein Interaction Data in WormBase

Protein interaction is an important data type to understand the biological function of proteins involved in the interaction, and helps researchers to deduce the biological nature of unknown proteins from the well-characterized functions of their interaction partners. High-throughput studies, coupled with the aggregation of individual experiments, provides a global 'snapshot' of the protein interactions occurring at all levels of biological processes or circumstances. This snapshot of the interaction network, the interactome, is important to understand the overall events up to the level of comparison between species or pathway simulation, or to find new factors yet undefined in the processes, or to add details to the biological processes and pathways. As of September 2018, WormBase (www.wormbase.org) (Lee et al. 2018) contains 28,279 physical protein-protein interactions for the roundworm Caenorhabditis elegans. Among these, 1500 protein-protein interactions have been curated by BioGRID as a collaboration with WormBase. Within the data set, 17,990 protein-protein interactions are unique, and 6,079 unique genes are involved in these interactions. In order to visualize the overall interaction map, a network diagram for all the unique interactions was generated by using the ‘Cytoscape’ program, version 3.6.1 (Shannon et al. 2003) (Figure 1A). These numbers represent a 108% increase in the number of interaction annotations since last year, 2017. These interaction data were curated from 1,251 peer-reviewed papers, which were selected from the literature by ‘Textpresso Central’ using automatic SVM (Support Vector Machine)-based text mining approaches (Fang et al. 2012; Müller et al. 2018) and manual verification. Compared to other databases providing C. elegans protein-protein interaction, WormBase now presents the largest data set, which has 1.72-fold more interaction annotations than IMEx (Orchard et al. 2012) and 4.51-fold more than BioGRID (Chatr-Aryamontri et al. 2017) (Figure 1B). Most significantly, WormBase now houses the complete protein interaction data from almost all of the C. elegans literature published from 1993 to 2018. The data sets presented at IMEx and BioGRID are annotated from 253 and 174 papers, respectively. All the physical interaction data in WormBase are supported by experimental evidence from original research papers. The statistics of the detection methods used as experimental evidence are shown in Figure 1C. The majority of the interaction data came from high throughput analysis such as large-scale yeast two-hybrid assays or mass-spectrometry, however, a significant portion of the data (13.1%) are supported by more direct detection methods using small-scale, low throughput methods such as co-immunoprecipitation or co-crystallography (Figure 1C). In WormBase, protein-protein interaction data can be found as a subclass of physical interaction data in the ‘Interactions widget’ on the gene report page. The Interactions widget provides all types of interaction data related to the gene of interest, such as physical, genetic, regulatory, and predicted interactions. All the interaction data are represented together in a graph created with ‘Cytoscape.js’ and a table. In the table, the gene names of interaction partners (bait-target) in the interaction are displayed along with the publication. The interaction details including the detection method are also captured in the summary and the remark field in the Interactions page. Users can query the data by using the search bar on the WormBase front page or download all the available data files from the WormBase FTP site (ftp://ftp.wormbase.org/pub/wormbase/releases/current-production-release/species/c_elegans/PRJNA13758 /annotation/c_elegans.PRJNA13758.WSXXX.interactions.txt.gz, where WSXXX is the database version release, like “WS267”). All the interaction data in WormBase will be available soon at the new information resource for multiple model organisms, the Alliance of Genome Resources (https://www.alliancegenome.org/). This site will integrate all the interaction data from human and from model organisms C. elegans, budding yeast (Saccharomyces cerevisiae), fruit fly (Drosophila melanogaster), zebrafish (Danio rerio), mouse (Mus musculus) and rat (Rattus norvegicus). Integrated views of interaction data from diverse model organisms will be extremely helpful to build interaction databases for species-to-species comparison, and to establish a disease model quickly based on the database. For the most efficient analysis of the interaction data in WormBase, we are now working on developing a new ‘Venn diagram tool’ and integrating the ‘Gene Set Enrichment Analysis tool’ (https://wormbase.org/tools/enrichment/tea/tea.cgi) into the Interactions widget. We will continue to curate other types of macro-molecular interactions including protein-DNA, protein-RNA and RNA-RNA interactions, as well as newly reported protein-protein interaction data to serve our research community

Text mining meets community curation: a newly designed curation platform to improve author experience and participation at WormBase

Biological knowledgebases rely on expert biocuration of the research literature to maintain up-to-date collections of data organized in machine-readable form. To enter information into knowledgebases, curators need to follow three steps: (i) identify papers containing relevant data, a process called triaging; (ii) recognize named entities; and (iii) extract and curate data in accordance with the underlying data models. WormBase (WB), the authoritative repository for research data on Caenorhabditis elegans and other nematodes, uses text mining (TM) to semi-automate its curation pipeline. In addition, WB engages its community, via an Author First Pass (AFP) system, to help recognize entities and classify data types in their recently published papers. In this paper, we present a new WB AFP system that combines TM and AFP into a single application to enhance community curation. The system employs string-searching algorithms and statistical methods (e.g. support vector machines (SVMs)) to extract biological entities and classify data types, and it presents the results to authors in a web form where they validate the extracted information, rather than enter it de novo as the previous form required. With this new system, we lessen the burden for authors, while at the same time receive valuable feedback on the performance of our TM tools. The new user interface also links out to specific structured data submission forms, e.g. for phenotype or expression pattern data, giving the authors the opportunity to contribute a more detailed curation that can be incorporated into WB with minimal curator review. Our approach is generalizable and could be applied to additional knowledgebases that would like to engage their user community in assisting with the curation. In the five months succeeding the launch of the new system, the response rate has been comparable with that of the previous AFP version, but the quality and quantity of the data received has greatly improved

Two new functions in the WormBase Enrichment Suite

Genome-wide experiments routinely generate large amounts of data that can be hard to interpret biologically. A common approach to interpreting these results is to employ enrichment analyses of controlled languages, known as ontologies, that describe various biological parameters such as gene molecular or biological function. In C. elegans, three distinct ontologies, the Gene Ontology (GO), Anatomy Ontology (AO), and the Worm Phenotype Ontology (WPO) are used to annotate gene function, expression and phenotype, respectively (Ashburner et al. 2000; Lee and Sternberg, 2003; Schindelman et al. 2011). Previously, we developed software to test datasets for enrichment of anatomical terms, called the Tissue Enrichment Analysis (TEA) tool (Angeles-Albores and Sternberg, 2016). Using the same hypergeometric statistical method, we extend enrichment testing to include WPO and GO, offering a unified approach to enrichment testing in C. elegans. The WormBase Enrichment Suite can be accessed via a user-friendly interface at http://www.wormbase.org/tools/enrichment/tea/tea.cgi. To validate the tools, we analyzed a previously published extracellular vesicle (EV)-releasing neuron (EVN) signature gene set derived from dissociated ciliated EV neurons (Wang et al. 2015) using WormBase Enrichment Suite based on the WS262 WormBase release. TEA correctly identified the CEM, hook sensillum and IL2 neuron as enriched tissues. The top phenotype associated with the EVN signature was chemosensory behavior. Gene Ontology enrichment analysis showed that cell projection and cell body were the most enriched cellular components in this gene set, followed by the biological processes neuropeptide signaling pathway and vesicle localization further down. The tutorial script used to generate the figure above can be viewed at: https://github.com/dangeles/TissueEnrichmentAnalysis/blob/master/tutorial/Tutorial.ipynb The addition of Gene Enrichment Analysis (GEA) and Phenotype Enrichment Analysis (PEA) to WormBase marks an important step towards a unified set of analyses that can help researchers to understand genomic datasets. These enrichment analyses will allow the community to fully benefit from the data curation ongoing at WormBase